Description

Integrate gpt-fast mixtral/moe example in TorchServe.

Type of change

• New feature (non-breaking change which adds functionality)

Feature/Issue validation/testing

Model inference(int8 quantized tensor paralleled Mixtral-MOE) logs provided in the attachment file.
mixtral_moe_logs.txt

Checklist:

• Did you have fun?
• Have you added tests that prove your fix is effective or that this feature works?
• Has code been commented, particularly in hard-to-understand areas?
• Have you made corresponding changes to the documentation?